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Abstract.  Online  behavioral  advertisers  track  users  across  websites,  often  without  users' 
knowledge.  Over  the  last  twelve  years,  the  online  behavioral  advertising  industry  has 
responded  to  the  resulting  privacy  concerns  and  pressure  from  the  FTC  by  creating  private  self- 
regulatory  bodies.  These  include  the  Network  Advertising  Initiative  (NAI)  and  an  umbrella 
organization  known  as  the  Digital  Advertising  Alliance  (DAA).  In  this  paper,  we  enumerate  the 
DAA  and  NAI  notice  and  choice  requirements  and  check  for  compliance  with  those 
requirements  by  examining  NAI  members'  privacy  policies  and  reviewing  ads  on  the  top  100 
websites.  We  also  test  DAA  and  NAI  opt-out  mechanisms  and  categorize  how  their  members 
define  opting  out.  Our  results  show  that  most  members  are  in  compliance  with  some  of  the 
notice  and  choice  requirements,  but  two  years  after  the  DAA  published  its  Self-Regulatory 
Principles,  there  are  still  numerous  instances  of  non-compliance.  Most  examples  of  non- 
compliance  are  related  to  the  "enhanced  notice"  requirement,  which  requires  advertisers  to 
mark  behavioral  ads  with  a  link  to  further  information  and  a  means  of  opting  out.  Revised 
October  7,  2011. 
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1  Introduction 

The  Federal  Trade  Commission  (FTC)  defines  online  behavioral  advertising  (OBA)  as  "the 
practice  of  tracking  consumers'  activities  online  to  target  advertising."1  The  FTC  has  been 
examining  ways  to  reduce  the  privacy  concerns  associated  with  OBA  for  over  a  decade. 

In  1999,  a  group  of  companies  engaging  in  OBA  announced  the  launch  of  a  self- 
regulatory  organization  called  the  Network  Advertising  Initiative  (NAI)  and  proposed  a  set  of 
principles  to  the  FTC.  In  a  July  2000  report  the  FTC  acknowledged  that  "the  NAI  principles 
present  a  solid  self-regulatory  scheme,"  but  nonetheless  recommended  legislation  to  provide  a 
basic  level  of  privacy  protection.2  This  legislation  was  never  enacted.3  The  NAI  published  its 


1  Federal  Trade  Commission,  Online  behavioral  advertising  moving  the  discussion  forward  to  possible  self- 
regulatory  principles,  http://www.ftc.gov/os/2007/12/P859900stmt.pdf  (December  2007,  retrieved  February 
2011) 

2  Federal  Trade  Commission,  Online  Profiling:  a  Report  to  Congress:  Part  2  Recommendations, 
http://www.ftc.gov/os/2000/07/onlineprofiling.pdf  (July  2000,  retrieved  February  2011) 

3  Federal  Trade  Commission,  Self-regulatory  principles  for  online  behavioral  advertising, 


principles  in  2001  and  and  revised  them  in  2008. 4  Today,  the  NAI  has  74  member  companies5 
and  offers  a  consumer  opt-out  service6  that  allows  consumers  "to  'opt  out'  of  the  behavioral 
advertising  delivered  by  our  member  companies."7 

As  the  FTC  began  examining  OBA  again  in  2009,  several  industry  organizations  with  an 
interest  in  OBA  (including  the  NAI)  formed  the  Digital  Advertising  Alliance  (DAA).8  One  of  the 
member  organizations  of  the  DAA  is  the  Interactive  Advertising  Bureau  (IAB),  which  lists  as  one 
of  its  "core  objectives"  to  "Fend  off  adverse  legislation  and  regulation."9  In  July  2009,  the  DAA 
published  its  own  set  of  requirements,  the  Self-Regulatory  Principles  for  Online  Behavioral 
Advertising,10  in  an  effort  to  avoid  an  FTC  push  for  new  legislation.11  The  self-regulatory 
program  based  on  the  DAA  principles  document  was  announced  in  October  2010.  According  to 
a  Better  Business  Bureau  announcement:12 

the  Principles  and  practices  represent  the  industry's  response  to  the  Federal 
Trade  Commission's  call  for  more  robust  and  effective  self-regulation  of  online 
behavioral  advertising  practices  that  would  foster  transparency,  knowledge  and 
choice  for  consumers. 

As  the  FTC  determines  what  to  do  next,  it  is  useful  to  evaluate  the  effectiveness  of 
industry  self-regulation  to  date.  In  this  paper,  we  focus  on  the  effectiveness  of  notice  and  opt- 
out,  and  quantify  DAA  and  NAI  member  compliance  with  these  self-regulatory  requirements. 
We  check  for  compliance  by  examining  websites  showing  advertisements,  advertising  network 
websites,  and  the  cookies  produced  by  the  DAA  and  NAI  opt-out  mechanisms. 

The  remainder  of  our  paper  is  organized  as  follows.  We  present  background  and  related 
work  in  Section  2.  Section  3  discusses  the  DAA  and  NAI  requirements  we  investigate.  We 
outline  our  methodology  in  Section  4  and  present  our  findings  in  Section  5.  Finally,  we  conclude 
with  a  discussion  in  Section  6. 

2  Background  and  Related  Work 


http://www.ftc.gov/os/2009/02/P085400behavadreport.pdf  (February  2009,  retrieved  February  2011) 

4  NAI,  2008  NAI  Principles:  The  Network  Advertising  Initiative's  Self-Regulatory  Code  of  Conduct, 
http://www.networkadvertising.org/networks/2008NAIPrinciplesfinalforWebsite.pdf  (2008) 

5  The  full  NAI  membership  list  is  available  online  at  http://www.networkadvertising.org/participating/ 

6  NAI,  Opt  Out  of  Behavioral  Advertising,  http://www.networkadvertising.org/managing/opt_out.asp 

7  Ibid. 

8  For  a  list  of  affiliated  organizations  see  http://www.aboutads.info/associations 

9  http://www.iab.net/about_the_iab 

10  Digital  Advertising  Alliance,  Self-Regulatory  Principles  for  Online  Behavioral  Advertising, 

http://www.aboutads.info/resource/download/seven-principles-07-01-09.pdf  (July  2009,  retrieved  January  2011) 

11  Davis  &  Gilbert  LLP,  Newly  Formed  Digital  Advertising  Alliance  Announces  Self-Regulatory  Program  For  Online 
Behavioural  Advertising,  http://www.dglaw.com/images_user/newsalerts/AdvMktngPromo_Behavioral- 
Advertising-Self-Regulatory-Program.pdf  (October  2010,  retrieved  February  2011) 

12  Better  Business  Bureau,  Major  marketing  /  media  trade  groups  launch  program  to  give  consumers  enhanced 
control  over  collection  and  use  of  web  viewing  data  for  online  behavioral  advertising,  Press  Release, 
http://www.newyork.bbb.org/article/major-marketing/media-trade-groups-launch-program-to-give-consumers- 
enhanced-control-over-collection-and-use-of-web-viewing-data-for-online-behavioral-advertising-22618  (October 
2010) 


Online  behavioral  advertising  is  a  form  of  advertising  in  which  advertising  networks  construct 
profiles  of  users  as  they  navigate  various  websites.13  The  purpose  of  this  tracking  is  to  present 
each  user  with  advertisements  expected  to  be  related  to  his  or  her  interests.14  HTTP  cookies  are 
the  primary  mechanism  for  executing  this  tracking,  though  it  is  possible  to  do  so  using  other 
technologies  such  as  JavaScript  cookies  or  Flash  Local  Shared  Objects  (LSOs). 

While  OBA  practitioners  claim  it  benefits  consumers,15  for  example  by  funding  website 
content,  the  FTC  notes  that  behavioral  advertising  raises  privacy  concerns  among  consumers, 
including:16 

...the  invisibility  of  the  data  collection  to  consumers;  the  shortcomings  of  current 
disclosures  about  the  practice;  the  potential  to  develop  and  store  detailed 
profiles  about  consumers;  and  the  risk  that  data  collected  for  behavioral 
advertising  --  including  sensitive  data  regarding  health,  finances,  or  children  — 
could  fall  into  the  wrong  hands  or  be  used  for  unanticipated  purposes. 

In  a  2009  study,  Turow  et  al.17  found  that  the  majority  of  American  adults  did  not  want 
advertisements  to  be  targeted  toward  their  interests,  even  if  done  anonymously.  They  also 
found  that  most  Americans  believe  a  law  should  require  advertisers  "to  immediately  delete 
information  about  their  internet  activity."  In  a  2010  study  by  McDonald  et  al.,  over  60%  of  more 
than  300  participants  saw  online  behavioral  advertising  as  "invasive."18 

Google  counsel  Pablo  Chavez  reported  on  Google's  OBA  opt-out  mechanism,  which  also 
allows  users  to  modify  their  interest  categories:19 

for  every  user  that  has  opted  out,  about  four  change  their  interest  categories 
and  remain  opted  in,  and  about  ten  do  nothing.  We  take  from  this  that  online 
users  appreciate  transparency  and  control,  and  become  more  comfortable  with 
data  collection  and  use  when  they  feel  it  happens  on  their  terms  and  in  full  view. 

Other  research  has  examined  online  self-regulatory  mechanisms.  McDonald  et  al. 
explored  the  cost  of  reading  online  privacy  policies.  They  discovered  that,  despite  being  a  self- 


Pam  Dixon,  The  Network  Advertising  Initiative:  Failing  at  Consumer  Protection  and  at  Self-Regulation,  in  World 
Privacy  Forum,  vol.  15,  p.  2009  (2007) 

14  Digital  Advertising  Alliance,  Flow  Interest  Based  Ads  Work,  http://www.aboutads.info/how-interest-based-ads- 
work/  (2010,  retrieved  February  2011) 

15  Randall  Rothenberg  et  al.,  Comments  of  the  Interactive  Advertising  Bureau  on  Online  Behavioral  Advertising 
Proposed  Principles,  http://www.ftc.gov/os/comments/behavioraladprinciples/080411interactiveadbureau.pdf 
(April  2008,  retrieved  February  2011) 

16  Supra  note  3 

17  Joseph  Turow  et  al.,  Americans  Reject  Tailored  Advertising  and  Three  Activities  that  Enable  It,  in  SSRN  eLibrary 
(September  29,  2009) 

18  Aleecia  McDonald  et  al.,  Americans'  Attitudes  About  Internet  Behavioral  Advertising  Practices,  in  Proceedings  of 
the  9th  Workshop  on  Privacy  in  the  Electronic  Society  (WPES)  (October  4,  2010) 

19  Pablo  L.  Chavez,  Re:  Privacy  roundtables,  http://www.ftc.gov/os/comments/privacyroundtable/544506- 
00134.pdf  (April  2010) 


regulatory  mechanism  designed  to  provide  users  with  notice,  website  privacy  policies  were  so 
verbose  and  densely  written  that  it  would  be  unreasonable  for  a  typical  user  to  read  the  privacy 
policy  of  each  website  visited.20  The  Platform  for  Privacy  Preferences  (P3P)  is  a  self-regulatory 
mechanism  for  websites  to  communicate  their  privacy  policies  to  user  agents  so  users  do  not 
have  to  read  them.21  Leon  et  al.  discovered  that  thousands  of  websites  use  P3P  compact 
policies  to  misrepresent  their  privacy  practices.22  Reay  et  al.  examined  P3P  policies  of  websites 
and  compared  them  with  the  legal  requirements  of  the  websites'  jurisdictions.  They  found  that 
websites  often  do  not  claim  to  follow  legal  privacy-related  requirements.23 

Prior  research  has  examined  the  usability  of  self-regulatory  privacy  mechanisms. 
McDonald  et  al.  found  that  only  11%  of  study  participants  were  able  to  determine  the  function 
of  the  NAI  opt-out  website.24  Further,  the  Annenberg  Public  Policy  Center  reports  that  many 
users  misunderstand  the  purpose  of  website  privacy  policies.  Their  report  states  that  over  half 
of  users  believe  that  a  website  having  a  privacy  policy  means  the  website  in  question  will  not 
share  data.25 

The  NAI  principles  document  highlights  the  importance  of  NAI  members  adhering  to  the 
principles:26 


NAI  members  believe  that  self  imposed  constraints  help  achieve  the 
balance  needed  to  preserve  consumer  confidence  in  the  use  of  this  revolutionary 
medium.  Even  where  there  is  reduced  privacy  impact  in  use  of  anonymous  or 
anonymized  data,  the  NAI  recognizes  that  consumers  will  only  trust  and  continue 
to  engage  with  advertisers  online  when  there  is  appropriate  deference  shown  to 
consumers'  concerns  about  the  privacy  of  their  websurfing  experience. 

The  NAI  states  that  they  rely  in  part  on  consumers  to  report  violations.27 

The  NAI's  2010  Annual  Compliance  Report  examines  the  34  NAI  companies  who  were 
members  at  the  start  of  2010.  The  report  found  that  "the  vast  majority  of  evaluated  member 
companies  met  their  compliance  obligations."  However,  the  report  also  indicated  that  there 
were  instances  of  opt-out  mechanisms  failing  and  failure  of  members  to  observe  requirements 
pertaining  to  "non-cookie  technologies."  There  was  also  a  member  using  sensitive  health- 
related  information  to  target  ads  without  opt-in  consent,  as  the  NAI  requires.  The  document 
states  that  the  NAI  is  working  on  policy  changes  to  address  their  findings.28 


20  Aleecia  McDonald  et  al.,  The  Cost  of  Reading  Privacy  Policies,  ISJLP  4,  543-897  (2009) 

21  Lorrie  Faith  Cranor,  Web  Privacy  with  P3P,  O'Reilly  &  Associates,  Inc.,  Sebastopol,  CA,  USA  (2002) 

22  Pedro  Giovanni  Leon  et  al.,  Token  Attempt:  The  Misrepresentation  of  Website  Privacy  Policies  Through  the 
Misuse  of  P3P  Compact  Policy  Tokens,  Tech.  Rep.  10-014,  Carnegie  Mellon  University,  CyLab  (2010) 

23  Ian  Reay  et  al.,  A  Large-Scale  Empirical  Study  of  P3P  Privacy  Policies:  Stated  Actions  vs.  Legal  Obligations.  ACM 
Trans.  Web  3(2),  1-34  (2009) 

24  Supra  note  18 

25  Joseph  Turow,  Americans  and  Online  Privacy:  The  System  is  Broken,  Annenberg  Public  Policy  Center,  University 
of  Pennsylvania,  Philadelphia,  PA,  USA  (2003) 

26  Supra  note  4 

27  NAI,  Network  Advertising  Initiative  FAQ:  What  do  I  do  if  I  Think  an  NAI  Member  Has  Violated  the  NAI  Privacy 
Principles?,  http://www.networkadvertising.Org/managing/faqs.asp#question_15 

28  Network  Advertising  Initiative,  2010  annual  compliance  report, 


The  NAI  compliance  report  also  indicates  that  one  NAI  member  withdrew  its 
membership.29  This  highlights  one  potential  problem  with  self-regulatory  organizations: 
members  who  do  not  wish  to  follow  the  self-regulation  process  can  simply  leave.  The  FTC 
expressed  this  concern  in  2000:30 

For  while  NAI's  current  membership  constitutes  over  90%  of  the  network 
advertising  industry  in  terms  of  revenue  and  ads  served,  only  legislation  can 
compel  the  remaining  10%  of  the  industry  to  comply  with  fair  information 
practice  principles.  Self-regulation  cannot  address  recalcitrant  and  bad  actors, 
new  entrants  to  the  market,  and  drop-outs  from  the  self-regulatory  program. 

The  "do  not  track"  mechanism  has  been  proposed  as  a  mechanism  to  allow  privacy- 
concerned  users  to  avoid  OBA  tracking,31  and  Jon  Leibowitz,  chairman  of  the  FTC,  has  expressed 
his  support.32  A  recent  release  of  Mozilla  Firefox  includes  a  "do  not  track"  feature  that  signals 
to  visited  websites  that  the  user  does  not  wish  to  be  tracked.33  Likewise,  Microsoft  Internet 
Explorer  9  includes  a  do  not  track  header  as  well  as  a  feature  called  "tracking  protection."34 
Google  has  also  introduced  a  Chrome  extension  which  enables  users  to  retain  persistent  opt- 
out  cookies.35  The  do-not-track  and  opt-out  mechanisms  both  rely  on  website  operators  to 
honor  user  preferences. 

3  DAA  and  NAI  Requirements  Investigated  in  this  Study 

In  this  section  we  discuss  the  DAA  and  NAI  principles  in  more  detail,  and  focus  on  the  notice 
and  choice  requirements  that  we  investigate  in  this  study. 

The  DAA  principles  are  contained  in  a  48-page  document,  published  in  2009. 36  This 
document  presents  seven  principles  along  with  commentary  and  implementation  guidance.  The 


http://www.networkadvertising.org/pdfs/2010_NAI_Compliance_Report.pdf  (February  2011,  retrieved  February 
2011) 

29  Ibid. 

30  Supra  note  2 

31  Peter  Eckersley,  What  Does  the  "Track"  in  "Do  Not  Track"  Mean?, 

https://www.eff.org/deeplinks/2011/02/what-does-track-do-not-track-mean  (February  2011,  retrieved  February 
2011) 

32  Jon  Leibowitz,  Preliminary  FTC  Staff  Privacy  Report:  Remarks  of  Chairman  Jon  Leibowitz, 
http://www.ftc.gov/speeches/leibowitz/101201privacyreportremarks.pdf  (December  2010,  retrieved  February 
2011) 

33  Mozilla,  Mozilla  Firefox  4  Beta,  Now  Including  "Do  Not  Track"  Capabilities, 

http://blog.mozilla.com/blog/2011/02/08/mozilla-firefox-4-beta-now-including-do-not-track-capabilities/ 
(February  2011,  retrieved  February  2011) 

34  Dean  Hachamovitch,  IE9  and  Privacy:  Introducing  Tracking  Protection, 

http://blogs.msdn.eom/b/ie/archive/2010/12/07/ie9-and-privacy-introducing-tracking-protection-v8.aspx 
(December  2010,  retrieved  February  2011) 

35  Sean  Harvey  et  al.,  Keep  Your  Opt-Outs,  http://googlepublicpolicy.blogspot.com/2011/01/keep-your-opt- 
outs.html  (January  2010,  retrieved  February  2011) 

36  Supra  note  10 


NAI  principles  are  contained  in  a  12-page  document,  last  revised  in  2008. 37  This  document 
describes  ten  principles,  and  does  not  include  the  more  extensive  commentary  and 
implementation  details  of  the  DAA  principles  document.  The  principles  documents  are  not 
exhaustive  lists  of  either  organization's  requirements,  as  we  discuss  below. 

We  examined  the  DAA  principles  document  to  determine  which  principles  lend 
themselves  to  compliance  checks  through  inspection  of  websites,  privacy  policies, 
advertisements,  and  cookies. 

•  Education  Principle:  The  DAA  must  maintain  a  central  educational  website  and 
provide  educational  ads.  The  educational  website  is  the  DAA  website  itself.38 
Checking  the  educational  ad  requirement  is  beyond  the  scope  of  this  study. 

•  Transparency  Principle:  Companies  must  provide  certain  information  on  their 
websites  and  in  ads.  We  check  this  principle  through  inspection  of  websites  and 
advertisements. 

•  Consumer  Control  Principle :  Companies  must  provide  a  mechanism  for  opting  out  of 
data  collection  for  online  behavioral  advertising.  We  check  this  through  examination 
of  opt-out  cookies. 

•  Security  Data  Principle-.  This  sets  forth  requirements  for  data  security.  We  are  unable 
to  check  this  because  it  pertains  to  internal  practices. 

•  Material  Changes  Principle:  Companies  must  obtain  consent  before  making  certain 
changes  to  their  practices.  We  are  unable  to  check  this  because  we  do  not  know 
when  companies  change  their  practices  or  what  steps  they  are  taking  to  obtain 
consent. 

•  Sensitive  Data  Principle:  Companies  must  take  additional  steps  when  handling 
sensitive  data.  We  cannot  check  this  because  we  do  not  know  what  data  a  given 
company  may  have  or  what  steps  they  take  to  handle  it. 

•  Accountability  Principle:  The  industry  must  develop  compliance  programs.  The 
Direct  Marketing  Association  and  Council  of  Better  Business  Bureaus  are  developing 
such  programs,39  but  a  review  of  these  programs  is  beyond  the  scope  of  this  paper. 

The  NAI  principles  document  contains  similar  principles  as  well  as  some  additional  principles 
that  are  not  relevant  to  our  analysis. 

The  DAA  Transparency  Principle  requires  that  companies  "give  clear,  meaningful,  and 
prominent  notice  on  their  own  Web  sites  that  describes  their  Online  Behavioral  Advertising 
data  collection  and  use  practices."  Companies  must  indicate  "the  types  of  data  collected 
online,"  "the  uses  of  such  data,"  a  "mechanism  for  exercising  choice"  about  data  collection  and 
use  for  online  behavioral  advertising,  and  "the  fact  that  they  adhere  to  these  principles."  The 
NAI  principles  also  require  the  above,  except  for  members  stating  that  they  adhere  to  the  DAA 
principles.  In  addition,  the  NAI  principles  require  that  a  member  disclose  what  online  behavioral 
advertising  activity  it  performs,  and  the  approximate  duration  for  which  it  retains  data  for 
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online  behavioral  advertising. 

The  DAA's  Transparency  Principle  includes  an  "enhanced  notice"  provision,  requiring 
that  websites  on  which  behavioral  advertising  data  is  collected  or  used  provide  a  "clear, 
meaningful  and  prominent  link"  to  a  "disclosure"  about  online  advertising.  This  link  must 
appear  on  every  page  "where  OBA  data  is  collected  or  used."  This  disclosure  must  contain 
either  a  list  of  advertisers  collecting  data  and  corresponding  links,  or  "a  link  to  an  industry- 
developed  Web  site"  containing  certain  information.  A  link  to  the  DAA  website  satisfies  this 
condition. 

The  DAA  principles  require  no  specific  icon,  and  none  is  depicted  in  the  document  itself; 
however,  it  does  mention  "common  wording  and  a  link/icon  that  consumers  will  come  to 
recognize."40  In  January  2010,  the  industry  introduced  the  "Power  I"  icon  to  denote  online 
behavioral  advertising.41  This  symbol  was  selected  based  on  the  results  of  a  research  study 
commissioned  by  the  Future  of  Privacy  Forum.42  Nine  months  later,  the  industry  announced  a 
new  "Advertising  Option  Icon."43  Both  the  original  and  new  icons  are  shown  in  Figure  1.  The  Ad 
Option  Icon  may  be  licensed  for  a  fee  from  the  DAA  (although  web  publishers  with  annual 
revenues  from  online  behavioral  advertising  of  less  than  $2,000,000  are  permitted  to  use  it  for 
free).44 


Enter 
ZIP  Code: 


AdGhaices  Q 


AtJChoices  [i> 


Figure  1:  A  Progressive  ad  (left)  and  a  Geico  ad  (right)  displaying  the  Power  I  and 
Advertising  Option  Icon,  respectively. 


The  DAA  Consumer  Control  principle  requires  that  companies  "provide  consumers  with 
the  ability  to  exercise  choice  with  respect  to  the  collection  and  use  of  data  for  Online 
Behavioral  Advertising  purposes."  This  must  be  available  from  one  of  a  number  of  locations, 
including  the  privacy  notice.  Likewise,  the  NAI  requires  that  its  members  using  non-personally 
identifiable  information  for  OBA  provide  users  with  an  opt-out  mechanism,  both  on  the 
member  website  and  on  the  NAI  website.  Further,  while  the  DAA  and  NAI  principles  documents 
do  not  mention  this,  the  NAI45  and  DAA46  both  require  that  opt-out  cookies  persist  for  at  least 
five  years. 

We  also  note  that  in  2009  the  FTC  narrowed  its  focus  to  third-party  behavioral 
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advertising.47  Thus,  the  DAA  considers  online  behavioral  advertising  to  occur  only  "across  non- 
Affiliate  Websites."48  The  DAA  states  that  the  principles  do  not  cover  "activities  of  First  Parties 
(Web  site  publishers  /  operators)  that  are  limited  to  their  own  sites  or  affiliated  sites  over  which 
they  exercise  direct  control."49  The  NAI  defines  online  behavioral  advertising  as  "third-party 
online  behavioral  advertising."50  Thus  a  website  can  still  track  and  target  ads  at  a  user  who  has 
opted  out  if  the  user  is  on  the  ad  network's  own  website. 

Based  on  this  analysis,  we  compiled  a  set  of  10  requirements  to  check  for  this  study. 
This  list  of  requirements  is  shown  in  Table  1. 

Table  1:  Summary  of  requirements  we  checked  in  this  study. 

Requirement  Source  How  Checked 


Privacy  notice  requirements 


Types  of  data  collected 

DAA+NAI 

NAI  member 

website 

Usage  of  collected  data 

DAA+NAI 

NAI  member 

website 

Presence  of  opt-out  mechanism 

DAA+NAI 

NAI  member 

website 

Adherence  to  DAA  principles 

DAA 

NAI  member 

website 

Behavioral  advertising  activities 

NAI 

NAI  member 

website 

How  long  data  is  retained 

NAI 

NAI  member 

website 

Enhanced  notice  requirement 

Advertisements  contain  enhanced  notice 

DAA 

Quantcast  top  100 

Opt-out  cookie  requirement 

Cookie  present  in  DAA  opt-out  mechanism 

DAA 

DAA  mechanism 

Cookie  present  in  NAI  opt-out  mechanism 

NAI 

NAI  mechanism 

Cookie  duration  is  at  least  five  years 

DAA+NAI 

Both  mechanisms 

The  IAB,  which  is  a  member  organization  of  the  DAA,  has  its  own  separate  code  of 
conduct.  At  the  time  of  this  writing,  this  document  contains  the  DAA  Principles  document 
verbatim,  as  well  as  a  section  on  monitoring  and  enforcement,  with  the  task  of  supervision 
given  to  the  Council  of  Better  Business  Bureaus.  The  IAB  has  also  posted  a  requirement  that 
their  members  become  compliant  with  this  code  by  August  29,  2011. 51 


Federal  Trade  Commission,  FTC  Staff  Revises  Online  Behavioral  Advertising  Principles, 
http://www.ftc.gov/opa/2009/02/behavad.shtm  (February  2009,  retrieved  February  2011) 
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4  Methodology 


In  February  and  March  2011,  we  analyzed  the  66  NAI  members  listed  on  the  NAI  website  as  of 
February  2011  for  compliance  with  the  requirements  in  Table  1.  To  see  if  NAI  member 
compliance  had  improved,  we  examined  the  74  NAI  member  websites  as  of  July  2011  again  in 
July  and  August  2011.  Then,  because  there  was  a  deadline  for  compliance  from  the  IAB  on 
August  29  2011,52  we  checked  member  websites  again  in  the  week  following  the  deadline  to 
see  whether  their  privacy  policies  had  been  changed  since  our  previous  check.  We  report  only 
the  results  of  the  final  check  for  each  member. 

We  examined  member  websites  for  the  privacy  notice  requirements  by  examining  the 
front  page  of  each  member's  website,  their  privacy  policy,  and  any  relevant  links  from  that 
policy.  We  considered  the  requirement  that  members  state  what  types  of  data  they  collect  for 
behavioral  advertising  satisfied  if  the  privacy  policy  provided  a  general  description  of  what  data 
is  collected  or  an  example.  We  considered  the  requirement  that  a  member  disclose  how  long  it 
retains  data  for  behavioral  advertising  satisfied  even  if  the  member  stated  it  retains  data 
indefinitely.  Flowever,  we  did  not  consider  the  requirement  satisfied  if  a  member  disclosed  only 
cookie  or  log  file  expiration  information. 

While  NAI  members  are  not  required  to  provide  their  own  definitions  of  opting-out,  we 
noted  whenever  a  member  chose  to  do  so.  We  categorized  these  members  as  defining  opting- 
out  to  mean  either  not  showing  targetted  ads;  collecting  some  less  data  fom  opted-out  users; 
no  longer  tracking  opted-out  users;  or  collecting  no  data  from  opted-out  users.  The  difference 
between  no  longer  tracking  users  and  collecting  no  data  from  users  at  all  is  that  in  the  former 
case,  aggregate  data  can  still  be  collected.  If  a  company  used  language  such  as  "we  no  longer 
collect  data  for  the  purpose  of  targetting  ads,"  we  counted  that  company  as  simply  not 
targetting  ads. 

We  examined  the  opt-out  cookies  from  the  DAA53  and  NAI54  opt-out  mechanisms,  in 
February  2011  and  in  August  2011.  We  checked  that  both  mechanisms  successfully  placed  opt- 
out  cookies  for  each  NAI  member,  checked  whether  the  two  mechanisms  provided  the  same 
cookies,  and  checked  whether  the  cookies  had  a  duration  of  at  least  five  years. 

In  mid-March  2011,  we  checked  compliance  with  the  enhanced  notice  requirement  of 
the  DAA  principles  by  inspecting  advertisements  on  websites  on  Quantcast's  February  2011  U.S. 
list  of  top  100  websites.55  We  repeated  this  in  Summer  2011;  we  checked  compliance  again  on 
the  same  websites  between  July  26  and  August  19  2011.  Then,  because  some  websites  might 
have  become  more  compliant  on  account  of  the  IAB  compliance  deadline  of  August  29  2011,56 
we  reexamined  any  website  which  had  ads  but  was  not  fully  compliant  during  our  previous 
check  between  August  31  and  September  2  2011. 

We  navigated  to  the  root  page  for  each  of  these  websites,  and  then  to  first  three  links 
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(from  top  to  bottom,  left  to  right)  pointing  to  non-search  pages  in  the  same  domain.  To  record 
which  advertising  networks  were  associated  with  each  page,  we  used  the  Firefox  web  browser 
with  the  TACO  add-on,57  which  enables  users  to  observe  the  advertising  networks  on  each 
website.  In  addition,  we  also  made  note  of  advertising  networks  that  were  explicitly  mentioned 
in  ad  disclosures. 

The  enhanced  notice  requirement  of  the  DAA  applies  only  to  behavioral  advertisements. 
It  is  nearly  impossible  to  determine  if  a  given  ad  is  behavioral  by  visual  inspection,  and  TACO 
indicates  whether  an  ad  network  is  present  on  a  website  but  not  whether  a  specific  ad  is 
behavioral.  In  order  to  remove  from  consideration  ads  that  were  unlikely  to  be  behavioral,  we 
excluded  ads  on  websites  where  TACO  did  not  recognize  an  ad  network.  In  addition,  we 
excluded  ads  that  the  DAA  requirements  likely  would  not  cover  because  they  appeared  (based 
on  our  judgement)  to  be  contextual  ads,  "based  on  the  content  of  the  Web  page  being  visited, 
a  consumer's  current  visit  to  a  Web  page,  or  a  search  query."58  For  example,  we  excluded  ads 
for  Comcast  products  on  comcast.com  and  ads  for  drugs  on  webmd.com. 

Industry  estimates  suggest  that  we  can  reasonably  assume  that  about  80%  of 
advertisements  we  encounter  are  behavioral.  Omar  Tawakol,  CEO  of  BlueKai,  stated  recently 
that  "eighty  percent  of  online  ads  rely  on  third-party  cookies  for  some  form  of  audience 
targeting."59  Likewise,  the  Interactive  Advertising  Bureau  stated  "in  an  IAB  survey  of  ad 
agencies  conducted  earlier  this  year,  we  found  that  80%  or  more  of  digital  advertising 
campaigns  were  touched  by  behavioral  targeting  in  some  way."60  On  the  other  hand,  industry 
representatives  distinguish  between  different  types  of  targeted  advertising,  and  Tawakol  has 
stated  that  "the  majority  of  third  party  cookie  use  for  targeting  actually  isn't  traditionally  called 
behavioral  advertising."61  It  is  not  entirely  clear  which  targeted  ads  and  third-party  cookies  are 
actually  subject  to  self-regulatory  requirements. 

At  each  website  on  the  Quantcast  top  100  list  we  did  the  following: 

1.  Create  a  new  Firefox  profile  (this  clears  cookies  and  the  cache)  and  clear  Flash  LSOs. 

2.  Copy  and  paste  the  URL  for  the  given  website  from  the  Quantcast  list. 

3.  Check  for  the  presence  of  non-contextual  ads  (ads  not  related  to  the  visited  website  or  the 
content  of  the  current  page). 

4.  If  there  are  non-contextual  ads,  check  them  for  compliance  with  the  DAA  principles  and 
record  the  tracking  websites  TACO  lists  for  the  page. 

5.  If  there  is  a  privacy  notice  associated  with  advertisements,  follow  the  link  and  record  its 
data. 
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6.  Repeat  steps  3  through  5  for  the  first  three  non-search  links  on  the  page. 

5  Results 

We  present  the  results  of  this  paper  in  four  parts.  In  Section  5.1,  we  present  the  evidence  of 
"enhanced  notice"  we  found  while  visiting  Quantcast's  top  100  websites.  In  Section  5.2,  we 
present  our  findings  for  compliance  with  "privacy  notice"  requirements.  We  evaluate  the  DAA 
and  NAI  opt-out  mechanisms  in  Section  5.3.  Finally,  in  Section  5.4  we  look  at  how  different  NAI 
members  define  opting  out.  For  all  requirements  we  check,  we  present  rates  of  compliance  and 
indicate  which  members  were  not  compliant. 

5.1  Enhanced  Notice  Requirement 

We  looked  for  non-contextual  ads  on  400  web  pages  across  100  websites.  In  Spring,  we  found 
164  pages  across  50  websites  that  contained  non-contextual  ads  and  were  monitored  by  NAI 
members  in  our  first  examination.  In  Summer,  we  found  155  pages  across  54  websites.  We 
focus  on  NAI  members  since  they  all  describe  themselves  as  engaged  in  OBA  and  are  required 
to  follow  both  DAA  and  NAI  requirements.  Using  TACO  to  determine  who  monitored  each  page, 
we  found  an  average  of  2.8  NAI  members  identified  per  page  in  Spring,  and  3.1  in  Summer. 

The  "enhanced  notice"  requirement  of  the  DAA's  Transparency  principle  requires  that 
notice  be  placed  on  the  same  page  where  behavioral  ads  appear.62  Using  the  methodology 
described  in  Section  4,  we  searched  for  evidence  of  this  notice  on  each  of  the  pages.  In  the 
Spring,  we  found  enhanced  notice  on  35%  of  these  pages.  In  the  Summer,  we  found  compliance 
on  50%  of  pages.  In  both  cases,  we  only  consider  pages  where  we  observed  non-contextual  ads 
that  were  tracked  by  an  NAI  member.  Since  we  expect  that  about  80%  of  advertisements  are 
behavioral,  this  represents  a  significant  gap  in  compliance  with  the  enhanced  notice 
requirement. 

While  we  looked  for  any  instance  of  enhanced  notice  on  a  webpage,  some  pages  did  not 
provide  this  notice  for  every  ad  on  the  page.  Specifically,  in  the  Spring,  we  found  45  pages  that 
provided  enhanced  notice  near  at  least  one  advertisement,  with  29  of  these  pages  providing 
enhanced  notice  near  every  ad  on  the  page.  In  addition,  12  pages  (on  three  websites)  provided 
notice  with  a  single  link  at  the  bottom  of  the  page.  In  the  Summer,  we  observed  54  pages  with 
enhanced  notice  near  at  least  one  advertisement,  of  which  31  pages  had  enhanced  notice  near 
all  advertisements.  46  pages  on  15  websites  provided  notice  with  a  single  link  at  the  bottom  of 
the  page.  We  are  unable  to  distinguish  between  those  ads  that  lacked  required  notice,  and 
those  that  are  not  behavioral  and  thus  are  not  required  to  provide  a  notice.  Links  found  at  the 
bottom  of  websites  do  not  list  the  advertising  providers  for  each  ad  on  the  page,  and  are 
arguably  not  very  prominent  since  they  may  require  a  large  amount  of  scrolling  to  find. 

Evidence  of  notice  was  also  inconsistent  across  pages  on  a  single  site.  Aside  from  the 
sites  that  provided  a  single  link  at  the  bottom  of  the  page,  seven  websites  displayed  enhanced 
notice  on  all  four  pages  that  we  visited,  with  an  additional  15  websites  providing  notice  on  at 
least  one  page  in  the  Spring.  In  the  Summer,  aside  from  websites  that  provided  enhanced 
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notice  with  a  link  at  the  bottom,  11  websites  provided  enhanced  notice  on  all  pages  we  visited, 
and  28  provided  enhanced  notice  on  at  least  one.  We  also  observed  a  mixing  of  notice  styles 
across  pages  on  a  single  site.  Table  2  lists  the  type  of  enhanced  notice  found  on  each  of  the  top 
websites  where  we  observed  non-contextual  ads. 


Table  2:  The  top  100  websites  for  the  U.S.  audience  as  ranked  by  Quantcast63  and  the  level  of 
compliance  with  the  enhanced  notice  requirement  that  we  observed.  Only  websites  on  which  we 
observed  non-contextual  ads  are  listed.  Note  that  mybloglog.com  (55  in  the  top  100)  is  excluded  from 
this  table.  It  did  not  show  non-contextual  ads  in  the  Spring,  and  in  the  Summer,  it  pointed  to  yahoo.com. 
Some  websites  appear  to  have  made  an  effort  toward  compliance,  without  being  entirely  compliant.  A 
website  marked  "Trying"  is  making  an  attempt  for  all  of  their  ads  to  be  compliant  by  placing  a  link  at  the 
bottom  of  the  web  page,  but  that  page  is  not  entirely  compliant. 


Rank 

Website 

Compliance 
Spring  2011 

Compliance 

Summer 

2011 

Enhanced  Notice  Observed 

3 

yahoo.com 

Fully 

Fully 

Ad.  Opt.  Icon,  Power  1,  Link  at  bottom 

4 

youtube.com 

N/A 

Fully 

Advertising  Option  Icon 

5 

msn.com 

Fully 

Fully 

Ad.  Opt.  Icon,  Power  1,  Link  at  bottom 

12 

aol.com 

No 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

14 

answers.com 

Some 

No 

Advertising  Option  Icon 

17 

ask.com 

Some 

Some 

Advertising  Option  Icon 

18 

ehow.com 

No 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

20 

about.com 

No 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

21 

myspace.com 

Some 

Some 

Power  1,  Ad.  Opt.  Icon 

22 

weather.com 

No 

Some 

Advertising  Option  Icon 

23 

mapquest.com 

Some 

No 

Advertising  Option  Icon 

26 

photobucket.com 

No 

No 

- 

27 

reference.com 

Some 

Some 

Power  1,  Ad.  Opt.  Icon 

31 

go.com 

N/A 

Some 

Link  at  bottom 

32 

huffingtonpost.com 

No 

No 

- 

34 

break.com 

No 

Fully 

Link  at  bottom 

35 

hulu.com 

N/A 

No 

- 

36 

comcast.net 

N/A 

Fully 

Link  near  ads 

38 

imdb.com 

Some 

None 

Advertising  Option  Icon 

39 

monster.com 

Some 

Some 

Advertising  Option  Icon 

41 

webmd.com 

N/A 

Fully 

Advertising  Option  Icon 

42 

pandora.com 

Some 

Some 

Advertising  Option  Icon 

45 

whitepages.com 

No 

Fully 

Link  at  bottom 

46 

associatedcontent.com 

Fully 

Fully 

Ad.  Opt.  Icon,  Power  1,  Link  at  bottom 

47 

cnn.com 

Fully 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

48 

flickr.com 

Fully 

N/A 

Link  near  ads 

50 

manta.com 

Fully 

Fully 

Advertising  Option  Icon 

54 

hubpages.com 

N/A 

Fully 

Power  1,  Ad.  Opt.  Icon 

56 

filmannex.com 

No 

No 

- 

57 

chinaontv.com 

No 

N/A 

- 

58 

digg.com 

No 

Some 

Advertising  Option  Icon 

59 

cnet.com 

Fully 

Fully 

Link  near  ads 

60 

yellowpages.com 

Fully 

Fully 

Power  1,  Link  at  bottom 

62 

washingtonpost.com 

Fully 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

64 

nytimes.com 

Trying 

Fully 

Ad.  Opt.  Icon,  Link  at  bottom 

66 

tripadvisor.com 

No 

N/A 

- 

67 

legacy.com 

Some 

Some 

Advertising  Option  Icon 

68 

evite.com 

No 

Some 

Advertising  Option  Icon 

63 
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69 

bbc.co.uk 

No 

Fully 

Link  at  bottom 

71 

people.com 

No 

Fully 

Link  at  bottom 

72 

chacha.com 

No 

Some 

Advertising  Option  Icon 

73 

tmz.com 

No 

Some 

Advertising  Option  Icon 

75 

drudgereport.com 

No 

No 

- 

77 

dailymotion.com 

N/A 

Some 

Link  near  ads 

79 

accuweather.com 

Trying 

Fully 

Ad.  Opt.  Icon,  Power  1,  Link  at  bottom 

80 

suitel01.com 

Some 

Some 

Advertising  Option  Icon 

81 

mtv.com 

Fully 

Fully 

Link  at  bottom 

83 

yelp.com 

No 

Some 

Advertising  Option  Icon 

86 

examiner.com 

Some 

No 

Power  1 

87 

wikia.com 

Some 

Fully 

Advertising  Option  Icon 

89 

squidoo.com 

Some 

Some 

Power  1,  Ad.  Opt.  Icon 

90 

merriam-webster.com 

Some 

Some 

Advertising  Option  Icon 

93 

weatherbug.com 

No 

No 

- 

94 

bizrate.com 

No 

No 

- 

96 

wunderground.com 

No 

Some 

Advertising  Option  Icon 

99 

twitpic.com 

Some 

Fully 

Advertising  Option  Icon 

100 

candystand.com 

No 

Fully 

Advertising  Option  Icon 

TACO  identified  trackers  from  23  NAI  members  in  the  Spring  and  28  in  the  Summer  on 
the  pages  we  examined.  When  TACO  found  NAI  members  tracking  a  page  that  had  non- 
contextual  ads,  we  expected  to  find  at  least  one  enhanced  notice.  In  the  Spring,  we  observed 
four  members  only  on  pages  with  enhanced  notice,  16  being  on  pages  with  and  without 
enhanced  notice,  and  three  only  on  pages  without  enhanced  notice.  In  the  Summer,  we  found 
10  members  only  on  pages  with  enhanced  notice,  7  members  on  pages  with  and  without 
enhanced  notice,  and  11  members  only  on  pages  without.  Table  3  presents  detailed  results  for 
each  NAI  member. 


Table  3.  Analysis  of  enhanced  notice  and  opt-out  cookies  for  NAI  members.  Enhanced  notice  data  was 
derived  by  examining  advertisements  on  the  Quantcast  top  100  U.S.  websites  gathered  in  Spring 
(March)  and  Summer  (late  August  to  early  September)  2011.  Blank  lines  indicate  no  instances  of  data- 
coilection.  Opt-out  mechanisms  were  tested  in  February,  March,  and  August  of  2011.  A  indicates  the 


member  was  not  in  the  NAI  during  collection.  Websites  marked  with  *  are  only  listed  as  NAI  members 
for  August.  Note  that  Batanga  does  not  have  its  own  opt-out  cookies. 


Name 

Pages  where 
member 

collects  data 

while  non- 

contextual  ad 
is  shown  (Spr. 

/  Sum.) 

Pages  where 
enhanced 

notice  was 
found  (Spr.  / 
Sum.) 

Number 
cookies  set  by 
DAA  opt-out 
(Feb.  /Mar./ 
Aug.) 

Number 
cookies  set  by 
NAI  opt-out 
(Feb.  /  Mar.  / 
Aug.) 

Do  its  DAA  and 

NAI  cookies 
match?  (Feb.  / 
Mar.  /  Aug.) 

[x+1] 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

24/7  Real  Media 

0/2 

0/0 

1/1/1 

1/1/4 

Yes  /  Yes  /  No 

33Across 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Adara  Media 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

AdBrite 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

AdChemy 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Adconion  Media  Group 

0/5 

0/5 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

*AddThis 

-/  19 

-  /  8 

-/-/ 1 

-/-/ 1 

-/-/Yes 

Adify 

1/3 

0/0 

1/1/1 

1/1/1 

Yes  /  Yes  /  No 

AdMeld 

4/26 

3/13 

0/0/1 

1/1/1 

No  /  No  /  Yes 

Aggregate  Knowledge 

1/1/2 

1/1/2 

Yes  /  Yes  /  Yes 

Akamai  Technologies 

2/2/3 

2/2/3 

Yes  /  Yes  /  Yes 

AOL  Advertising 

57/47 

20/24 

4/4/7 

6/7/7 

No  /  No  /  Yes 

*Aperature 

-/-/ 1 

-/-/ 1 

-/-/Yes 

Atlas 

0/5 

0/5 

1/1/0 

2/2/1 

No /No /No 

AudienceScience 

39/48 

11/25 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Batanga 

0/0/0 

0/0/0 

NA  /  NA  /  NA 

Bizo 

4/4/5 

4/4/5 

Yes  /  Yes  /  Yes 

BlueKai 

13/17 

11/11 

2/2/1 

2/2/1 

No  /  No  /  Yes 

*BrightRoll 

-/-/ o 

-/-/ 1 

-/-/No 

Brilig 

1/0/1 

1/1/1 

Yes  /  No  /  Yes 

Burst  Media 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Buysight 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Casale  Media 

21/5 

3/1 

1/1/1 

1/1/1 

Yes /Yes  /  Yes 

^Cognitive  Match 

-/-/ 0 

-  /  -  /  4 

-/-/No 

Collective 

20/9 

9/8 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Criteo 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

*Cross  Pixel  Media 

-/-/ 1 

-/-/ 1 

-/-/Yes 

DataLogix 

1/0 

0/0 

2/2/2 

2/2/2 

Yes/  Yes  /  Yes 

DataXu 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Datonics 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Dedicated  Networks 

0/1 

0/1 

0/1/1 

1/1/1 

No  /  Yes  /  Yes 

Dotomi 

6/0 

3/0 

2/2/1 

2/2/1 

Yes  /  Yes  /  Yes 

Epic  Marketplace 

2/0 

2/0 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

eXelate 

2/2/2 

2/2/2 

Yes  /  Yes  /  No 

FetchBack 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Glam  Media 

0/1/1 

1/1/1 

No  /  Yes  /  Yes 

Google 

127  /  148 

43/74 

2/1/6 

1/2/1 

No /No /No 

1-Behavior 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

interCUCK 

3/11 

3/5 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Invite  Media 

11/11/2 

11/11/11 

Yes  /  Yes  /  No 

Lotame 

4/1 

0/0 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

MAGNETIC 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

*MaxPoint  Interactive 

-/-/ o 

-/-/ 1 

-/-/No 

*Media  Innovation  Group 

-/I 

-/I 

-/-/o 

-  /  -  /  3 

-/-/No 

Media6Degrees 

7/3 

1/3 

1/1/1 

1/1/1 

Yes  /  No  /  No 

MediaMath 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

*MediaMind 

-  /  4 

-  /  4 

-/-/ 0 

-/-/ 1 

-/-/No 

Mediaplex 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Microsoft 

4/4 

4/4 

4/4/1 

4/4/4 

Yes  /  Yes  /  No 

Mindset  Media 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Netmining 

1/1/1 

1/1/0 

Yes  /  Yes  /  No 

OwnerlQ 

0/0/1 

1/1/1 

No  /  No  /  Yes 

*Pulse360 

-  /  4 

-  /  4 

-/-/ 1 

-/-/ 1 

-/-/Yes 

Quantcast 

101  /  89 

30/38 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

*RadiumOne 

-/-/ 1 

-/-/ 1 

-/-/Yes 

Red  Aril 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Rich  Relevance 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Rocket  Fuel 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

SpecificMEDIA 

5/0 

5/0 

3/3/3 

3/3/3 

Yes  /  Yes  /  Yes 

TARGUSinfo 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

The  Fox  Audience  Network 

6/5 

3/3 

3/3/5 

3/3/3 

Yes  /  Yes  /  No 

TidalTV 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Tribal  Fusion 

13/12 

4/2 

0/0/1 

1/1/1 

No  /  No  /  Yes 

*Tru  Effect 

-/-/ o 

-/-/ 1 

-/-/No 

Tumri 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Turn 

0/5 

0/5 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Undertone  Networks 

2/2/2 

2/2/2 

Yes  /  Yes  /  Yes 

ValueClick  Media 

0/1 

0/1 

2/2/1 

2/2/2 

Yes  /  Yes  /  No 

Vibrant  In-Text  Solutions 

2/4 

1/2 

1/1/1 

1/1/1 

Yes  /  Yes  /  No 

Wall  Street  on  Demand 

1/1/2 

1/1/2 

Yes  /  Yes  /  No 

XGraph 

3/0 

1/0 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

Yahoo! 

28/21 

8/13 

2/2/3 

2/2/5 

No /No /No 

YuMe 

1/1/1 

1/1/1 

Yes  /  Yes  /  Yes 

In  the  Summer,  over  the  74  instances  of  enhanced  notice  that  identified  the  ad  provider, 
we  noted  17  NAI  members.  We  noted  Google  most  often,  with  41  instances.  The  next  most 
common  member  was  Yahoo!,  with  7  instances. 

As  shown  in  Table  2,  we  observed  a  considerable  increase  in  compliance  between  Spring 
and  Summer,  with  many  improvements  being  made  right  around  the  lAB's  August  29  deadline. 
Of  the  100  websites  we  examined,  49  had  at  least  one  non-contextual  ad  during  both  the  Spring 
and  Summer  observations.  Of  these,  twenty-five  (51%)  retained  the  same  status,  while  twenty 
(41%)  improved.  In  the  Summer,  of  the  54  websites  that  had  ads,  44  (82%)  were  at  least 
somewhat  compliant  with  the  Enhanced  Notice  requirement,  and  26  (44%)  were  fully 
compliant.  Much  of  this  new  compliance  is  acheived  through  putting  ad  notice  links  at  the 
bottom  of  pages;  only  three  websites  used  this  technique  in  our  Spring  observation,  while  17 
did  in  the  Summer. 

Notably,  much  of  the  enhanced  notice  appeared  to  be  driven  by  advertisers  (i.e.  the 


companies  that  purchase  ads)  rather  than  by  NAI  members.  For  example,  almost  all  of  the 
Verizon  ads  we  saw  had  enhanced  notice,  even  though  they  came  from  many  different  ad 
providers,  including  AOL  Advertising,  Collective,  Google,  interCLICK,  and  Traffic  Marketplace. 
This  suggests  that  some  online  advertising  buyers  are  interested  in  providing  notice  and  choice 
to  their  customers.  This  also  means  that  a  website  using  symbols  on  ads  for  compliance  might 
have  a  varying  level  of  compliance  as  a  function  of  the  ads  being  served.  On  the  other  hand,  a 
website  correctly  using  a  link  at  the  bottom  of  the  page  will  be  consistently  compliant,  although 
with  a  less  prominent  notice. 

5.2  Privacy  Notice  Requirement 

We  checked  the  privacy  policies  of  the  66  NAI  members  for  compliance  with  the  privacy  notice 
requirements  from  Table  1  in  February  2011.  Audience  Science  and  Rocket  Fuel  were  the  only 
NAI  members  that  stated  that  they  adhere  to  the  DAA  principles,  and  thus  the  only  members 
fully  compliant  with  the  privacy  notice  requirements  we  checked.  Excluding  the  requirement  to 
mention  adherence  to  the  DAA  principles,  55  members  (83%)  were  compliant  with  the  privacy 
notice  requirements.  We  repeated  our  examination  after  the  August  IAB  deadline,  as  described 
in  Section  4.  There  are  now  74  NAI  members,  and  18  state  adherence  to  the  DAA  principles.  Of 
these,  14  have  changed  their  privacy  policies  to  indicate  adherence,  and  two  are  new  NAI 
members. 

All  NAI  members  mention  their  OBA  activities,  how  collected  data  is  used,  and  all 
provide  an  opt-out  mechanism.  It  is  worth  noting,  however,  that  as  of  our  Summer  check,  both 
Excelate  and  Tumri  provide  dead  opt-out  links  on  their  privacy  policies.  All  except  Fox  Audience 
Network  stated  what  types  of  data  they  collect  for  behavioral  advertising  during  our  Spring 
examination.  In  the  summer  examination  all  members  stated  what  types  of  data  they  collect  for 
behavioral  advertising.  Only  56  of  66  members  (85%)  in  the  spring  and  52  of  74  members  (84%) 
in  the  Summer  stated  how  long  they  retain  their  data  collected  for  behavioral  advertising.  Many 
members  mention  cookie  or  log  file  expiration  but  this  does  not  address  the  data  collected 
from  observing  cookies  or  analyzing  log  files.  Privacy  notice  requirement  compliance  for  each 
NAI  member  is  presented  in  Table  4. 


Table  4.  NAI  Member  Privacy  notice  compliance  for  February  2011  and  August  2011.  A  "No"  indicates 
that  notice  was  not  found  in  the  member's  privacy  policy.  If  the  value  is  the  same  for  February  and 
August,  it  is  listed  once.  If  there  is  a  change  between  February  and  August,  it  is  listed  as  FebruaryValue  - 
AugustValue.  Websites  marked  with  *  are  only  listed  as  NAI  members  for  August. 


Name 

Types  of 
data 

collected 

How  data 

will  be  used 

Adherence 

to  DAA 
Principles 

How  long 
data  will 

be  retained 

[x+1] 

Yes 

Yes 

No 

No1 

24/7  Real  Media 

Yes 

Yes 

No-Yes 

Yes 

33Across 

Yes 

Yes 

No 

Yes 

Adara  Media 

Yes 

Yes 

No 

Yes 

AdBrite 

Yes 

Yes 

No 

Yes 

AdChemy 

Yes 

Yes 

No 

Yes 

Adconion  Media  Group 

Yes 

Yes 

No 

Yes 

*AddThis 

Yes 

Yes 

Yes 

Yes 

Adify 

Yes 

Yes 

No 

Yes 

AdMeld 

Yes 

Yes 

No 

Yes 

Aggregate  Knowledge 

Yes 

Yes 

No 

Yes 

Akamai  Technologies 

Yes 

Yes 

No 

Yes 

AOL  Advertising 

Yes 

Yes 

No-Yes 

Yes 

*Aperature 

Yes 

Yes 

No 

No1 

Atlas 

Yes 

Yes 

No 

Yes 

AudienceScience 

Yes 

Yes 

Yes 

Yes 

Batanga 

Yes 

Yes 

No 

Yes 

Bizo6 

Yes 

Yes 

No-Yes 

Yes1 

BlueKai 

Yes 

Yes 

No-Yes 

Yes 

*  BrightRoll 

Yes 

Yes 

No 

No 

Brilig 

Yes 

Yes 

No 

Yes 

Burst  Media 

Yes 

Yes 

No 

Yes 

Buysight 

Yes 

Yes 

No 

Yes 

Casale  Media 

Yes 

Yes 

No-Yes 

No2 

“"Cognitive  Match 

Yes 

Yes 

No 

Yes 

Collective 

Yes 

Yes 

No-Yes 

Yes 

Criteo 

Yes 

Yes 

No 

Yes 

*Cross  Pixel  Media 

Yes 

Yes 

No 

Yes 

DataLogix 

Yes 

Yes 

No-Yes 

No3 

Datonics 

Yes 

Yes 

No 

Yes 

DataXu 

Yes 

Yes 

No-Yes 

Yes 

Dedicated  Networks 

Yes 

Yes 

No 

No1 

Dotomi 

Yes 

Yes 

No 

Yes 

Epic  Marketplace 

Yes 

Yes 

No 

Yes 

eXelate 

Yes 

Yes 

No 

Yes 

FetchBack 

Yes 

Yes 

No 

Yes 

Glam  Media 

Yes 

Yes 

No 

Yes 

Google 

Yes 

Yes 

No 

No 

1-Behavior 

Yes 

Yes 

No 

Yes 

interCLICK 

Yes 

Yes 

No 

Yes 

Invite  Media 

Yes 

Yes 

No 

Yes 

Lota  me 

Yes 

Yes 

No 

Yes 

MAGNETIC 

Yes 

Yes 

No 

Yes 

*MaxPoint  Interactive 

Yes 

Yes 

No 

Yes 

*Media  Innovation  Group 

Yes 

Yes 

Yes 

Yes 

Media6Degrees 

Yes 

Yes 

No 

Yes 

MediaMath 

Yes 

Yes 

No-Yes 

Yes 

*MediaMind  Technologies 

Yes 

Yes 

No 

Yes 

Mediaplex 

Yes 

Yes 

No 

Yes 

Microsoft 

Yes 

Yes 

No 

No1'4 

Mindset  Media 

Yes 

Yes 

No 

Yes 

Netmining 

Yes 

Yes 

No 

Yes 

OwnerlQ 

Yes 

Yes 

No 

No1 

*Pulse360 

Yes 

Yes 

No 

Yes 

Quantcast 

Yes 

Yes 

No-Yes 

Yes 

*RadiumOne 

Yes 

Yes 

No 

Yes 

Red  Aril 

Yes 

Yes 

No 

Yes 

richrelevance 

Yes 

Yes 

No-Yes 

Yes 

Rocket  Fuel 

Yes 

Yes 

No-Yes 

Yes 

SpecificMEDIA 

Yes 

Yes 

No 

Yes 

TARGUSinfo 

Yes 

Yes 

No 

No1 

The  Fox  Audience  Network 

No5  -Yes 

Yes 

No-Yes 

Yes-No 

TidalTV 

Yes 

Yes 

No 

Yes 

Tribal  Fusion 

Yes 

Yes 

No 

Yes 

*Tru  Effect 

Yes 

Yes 

No 

No 

Tumri 

Yes 

Yes 

No 

Yes 

Turn 

Yes 

Yes 

No 

Yes 

Undertone  Networks 

Yes 

Yes 

No-Yes 

Yes 

ValueClick  Media 

Yes 

Yes 

No 

Yes 

Vibrant  In-Text  Solutions 

Yes 

Yes 

No 

Yes 

Wall  Street  on  Demand 

Yes 

Yes 

No 

Yes 

Xgraph 

Yes 

Yes 

No 

Yes 

Yahoo! 

Yes 

Yes 

No-Yes 

No2 

YuMe 

Yes 

Yes 

No 

Yes 

1  Notice  only  mentions  cookie  expiration. 

2  Notice  only  mentions  log  file  retention. 

3  Notice  only  mentions  cookie  expiration  and  log  file  retention. 

4  Retention  information  found  in  a  blog  post,  not  in  prominent  location. 

5  Notice  explains  that  "non-personally  identifiable  information  obtained  from  cookies,  web  beacons, 


and/or  similar  monitoring  technologies"  is  collected,  but  the  types  of  data  are  not  specified. 

6  We  were  notified  that  Bizo's  privacy  policy  became  compliant  with  the  data  retention  requirement  on 
March  16,  2011. 


5.3  Choice  Requirement 

We  evaluated  the  NAI  and  DAA  opt-out  mechanisms  in  February  and  March  2011,  with  26  days 
between  checks.  We  used  Microsoft  Windows  with  Chrome  9.0.597,  Internet  Explorer 
8.0.6001.19019,  and  Firefox  3.6.13  browsers;  the  March  evaluation  used  Chrome  10.0.648.  We 
also  conducted  the  evaluation  in  August  2011,  using  Chrome  13.0.782.107,  Internet  Explorer 
8.0.6001.18702,  and  Firefox  5.0.1.  The  DAA  mechanism  reported  that  it  failed  to  set  an  opt-out 
cookie  for  one  company  when  we  tested  it  in  February  with  each  browser  -  in  all  three  cases, 
one  company  failed,  but  surprisingly  it  was  not  the  same  company  each  time.  On  Chrome  and 
Internet  Explorer,  the  DAA  mechanism  was  unable  to  set  the  opt-out  cookie  for  AOL 
Advertising,  the  third  most  pervasive  online  advertiser.64  On  Firefox,  the  mechanism  failed  for 
Audience  Science.  The  NAI  mechanism  was  able  to  set  all  opt-out  cookies  successfully. 

In  March,  we  retested  the  DAA  mechanism  and  found  the  Invite  Media  opt-out  cookie 
could  not  be  set  on  Chrome,  but  the  mechanism  worked  with  the  other  browsers.  In  August,  we 
successfully  used  Chrome  to  opt-out  from  NAI  members  using  the  DAA  mechanism.  Firefox 
failed  to  opt  out  of  TARGUSinfo,  and  Internet  Explorer  failed  to  opt  out  of  Microsoft 
Advertising.  On  the  NAI  website.  Chrome  and  Firefox  opted  out  successfully  from  all  members. 
Internet  Explorer  failed  for  Adconion,  Batanga,  BrightRoll,  Cognitive  Match,  Collective,  Media 
Innovation  Group,  MediaMind,  Microsoft  (Atlas  Technology),  TARGUSinfo,  and  TruEffect. 

We  also  observed  that  the  two  opt-out  mechanisms  sometimes  set  different  cookies, 
and  some  opt-out  cookies  changed  from  February  to  March  to  August.  Even  when  both 
mechanisms  set  cookies  for  the  same  advertiser,  they  did  not  always  agree  on  the  content  of 
the  cookie  or  the  number  of  cookies  that  were  set.  For  example,  the  NAI  mechanism  set  four 
cookies  for  the  domain  adsonar.com,  a  serving  domain  of  AOL  Advertising.  These  cookies  had 
the  names:  TData,  TData2,  atdemo,  and  atdemo2.  For  the  same  domain,  the  DAA  mechanism 
set  a  single  cookie  with  the  name  oo_flag.  This  did  not  change  between  February  and  March. 
Since  these  mechanisms  were  not  consistent,  users  might  have  needed  to  use  both 
mechanisms  to  opt-out.  However,  in  August,  the  adsonar  cookies  for  the  DAA  and  NAI  now 
match.  Summary  results  for  each  NAI  member  can  be  found  in  Table  3. 


64  Stephanie  Flosi,  comScore  Media  Metrix  ranks  top  50  U.S.  web  properties  for  October  2010,  Press  Release, 
http://comscore.eom/Press_Events/Press_Releases/2010/ll/comScore_Media_Metrix_Ranks_Top_50_U.S._Web 
_Properties_for_October_2010  (October  2010,  retrieved  November  2010) 


We  also  checked  opt-out  cookies  to  be  sure  that  they  persist  for  five  years,  in  keeping 
with  the  DAA65  and  NAI66  requirements.  Since  multiple  opt-out  cookies  can  be  set  for  a  single 
domain,  we  considered  a  domain  to  be  compliant  if  at  least  one  of  the  opt-out  cookies  had  a 
duration  of  at  least  five  years.  Three  domains:  adsonar.com,  advertising.com,  and 
invitemedia.com,  were  not  compliant  when  their  cookies  were  set  with  the  NAI  mechanism  in 
February.  Only  invitemedia.com  was  non-compliant  when  using  the  DAA  mechanism.  This 
shows  another  dimension  of  inconsistency  between  the  two  mechanisms.  In  March, 
invitemedia.com  became  compliant  with  both  mechanisms,  but  adsonar.com  and 
advertising.com  were  still  not  compliant.  In  August,  however,  all  cookies  were  compliant  with 
the  five  year  requirement. 

The  DAA  and  NAI  opt-out  mechanisms  do  not  function  in  the  Apple  Safari  browser  with 
default  settings.  Safari  blocks  third-party  cookies  from  being  set;  a  cookie  for  a  given  domain 
can  be  set  only  when  a  user  navigates  there.  A  user  who  navigates  to  an  advertising  network 
website  may  subsequently  be  tracked  by  that  network  across  other  websites  and  is  unable  to 
use  either  mechanism  to  opt  out  of  this  tracking.  To  confirm,  we  navigated  to  various  websites 
with  Safari  5.0.3  and  then  attempted  to  use  the  NAI  opt-out  mechanism.  Several  advertising 
networks  had  placed  tracking  cookies  on  our  computer,  but  we  were  unable  to  opt-out  from 
them  using  the  mechanism. 


5.4  Definitions  of  Opting  Out 

The  DAA  requires  that  its  members  provide  "users  of  Web  sites  at  which  data  is  collected  and 
used  for  Online  Behavioral  Advertising  purposes  the  ability  to  choose  whether  data  is  collected 
and  used  for  such  purposes."  The  DAA  website  says  that  opting  out  will  not  stop  data  collection, 
but  will  stop  delivery  of  ads  based  on  preferences.67  Consistent  with  the  DAA's  definition,  the 
NAI  defines  opting  out  as  follows:68 

Opt  out  of  OBA  means  that  a  consumer  is  provided  an  opportunity  to  exercise  a 
choice  to  disallow  OBA  with  respect  to  a  particular  browser.  If  a  consumer  elects 
to  opt  out  of  non-PII  OBA,  collection  of  non-PII  data  regarding  that  consumer's 
browser  may  only  continue  for  non-OBA  purposes,  such  as  ad  delivery  & 
reporting. 

Still,  as  of  our  Summer  check,  69  of  74  NAI  members  provide  their  own  definitions  of 
opt-out,  sometimes  going  beyond  the  NAI  and  DAA  requirements.69  For  example,  AdBrite  states 
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68  Supra  note  4 

69  The  members  that  did  not  define  opting  out  are  Aggregate  Knowledge,  Atlas,  Dotomi,  MediaMath,  The  Fox 
Audience  Network 


that  it  will  delete  prior  data  when  a  user  opts  out.  Bizo  indicates  it  will  stop  collecting  uniquely 
identifiable  data.  Whereas,  Addthis  just  states  that  it  will  no  longer  target  advertisements. 

Of  those  69  websites  that  define  opting  out,  42  indicate  collecting  less  or  no  data  or  no 
longer  tracking  the  user,  and  35  of  those  42  indicate  collecting  no  data  or  not  tracking  the  user. 
The  other  27  members  that  define  opting  out  indicate  only  that  opting  out  would  entail  not 
seeing  targeted  ads,  which  is  consistent  with  the  minimum  requirements  of  the  DAA  and  NAI. 
Three  of  these  members  explicitly  state  that  information  collection  would  continue.  These 
findings  are  detailed  in  Table  5. 


Table  5.  Categorized  definitions  of  opting  out  based  on  NAI  members'  privacy  policies.  Only  members 
that  defined  opting  out  are  included  in  this  table. 


NAI  Member 

Stated  Policy 

[x+1] 

N/A  -  Stop  tracking4 

24/7  Real  Media 

Collect  no  data1  -  Don't  target  ads 

33Across 

Collect  no  data 

Adara  Media 

Don't  target  ads 

AdBrite 

Collect  less  data3 

AdChemy 

Collect  no  data 

Adconion  Media  Group 

Don't  target  ads 

*AddThis 

Collect  no  data 

Adify 

Stop  tracking 

AdMeld 

Collect  no  data 

Akamai 

Don't  target  ads5 

AOL  Advertising 

Don't  target  ads 

*  Aperture 

Collect  no  data4 

AudienceScience 

Collect  no  data 

Batanga 

Collect  no  data 

Bizo 

Stop  tracking 

BlueKai 

Collect  less  data 

*  BrightRoll 

Don't  target  ads 

Brilig 

Collect  no  data 

Burst  Media 

N/A  -  Stop  tracking 

Buysight 

Collect  no  data 

Casale  Media 

Stop  tracking 

*Cognitive  Match 

Collect  no  data 

Collective 

Collect  no  data 

Criteo 

Don't  target  ads 

*Cross  Pixel  Media 

Collect  no  data 

DataLogix 

Don't  target  ads  -  Collect  no  data 

DataXu 

N/A  -  Don't  target  ads 

Datonics 

Collect  no  data2 

Dedicated  Networks 

Collect  no  data 

Epic  Marketplace 

Don't  target  ads 

eXelate 

Don't  target  ads  -  Collect  no  data 

Fetch  Back 

Don't  target  ads 

Glam  Media 

Stop  tracking1  -  Don't  target  ads 

Google 

Collect  less  data 

1-Behavior 

Don't  target  ads 

interCLICK 

Stop  tracking 

Invite  Media 

Don't  target  ads4 

Lotame 

Don't  target  ads 

MAGNETIC 

Don't  target  ads  -  Collect  less  data1 

*MaxPoint  Interactive 

Don't  target  ads 

*Media  Innovation  Group 

Collect  no  data 

Media6Degrees 

Don't  target  ads 

*MediaMind  Technologies 

Stop  tracking 

Mediaplex 

Stop  tracking4 

Microsoft 

Don't  target  ads5 

Mindset  Media 

Stop  tracking 

Netmining 

Collect  no  data  -  Don't  target  ads 

OwnerlQ 

Collect  no  data 

*Pulse360 

Don't  target  ads 

Quantcast 

Don't  target  ads  -  Collect  no  data 

*RadiumOne 

Collect  no  data3 

Red  Aril 

Collect  no  data2  -  Don't  target  ads 

richrelevance 

Don't  target  ads 

Rocket  Fuel 

Stop  tracking 

SpecificMEDIA 

Don't  target  ads 

TARGUSinfo 

Don't  target  ads 

The  Fox  Audience  Network 

Don't  target  ads5  -  N/A 

TidalTV 

Don't  target  ads 

Tribal  Fusion 

Stop  tracking 

*Tru  Effect 

Collect  no  data 

Tumri 

Don't  target  ads  -  Collect  less  data 

Turn 

Don't  target  ads  -  Collect  less  data 

Undertone  Networks 

Collect  less  data1 

ValueClick  Media 

Don't  target  ads 

Vibrant  In-Text  Solutions 

Collect  no  data 

Wall  Street  on  Demand 

Stop  tracking 

XGraph 

N/A  -  Collect  no  data 

Yahoo! 

N/A  -  Don't  target  ads 

YuMe 

Don't  target  ads 

1  Opt-out  definition  mentions  cookies  only;  we  assume  other  tracking  technologies  are  not  used. 

2  The  opt-out  cookie  is  defined  as  indicating  a  preference;  we  assume  this  preference  will  be  respected. 

3  Prior-held  data  will  be  deleted. 

4  The  opt-out  cookie  will  block  the  placement  of  other  cookies  from  this  advertiser. 

5  Explicitly  stated  that  data  collection  will  continue. 


5.5  Specific  Privacy  Policy  Notes 


There  are  several  cases  in  which  an  NAI  member  states  in  its  privacy  policy  that  a  previous  opt- 
out  effort  by  a  user  may  have  become  invalid.  According  to  the  privacy  policy  of  Akami,70  which 
purchased  aCerno,  "Due  to  technical  issues,  if  you  opted  out  of  targeted  advertising  by  acerno, 
your  choice  may  not  have  been  properly  saved  and  recognized."  Likewise,  according  to  the 
Dedicated  Networks  privacy  policy,  "As  a  result,  if  you  opted  out  of  targeted  advertising  by 
Dedicated  Networks  prior  to  January  2011,  your  choice  may  no  longer  be  fully  effective."71 
According  to  the  privacy  policy  of  Undertone,  "If  you  opted  out  of  targeted  advertising  between 
August  2008  and  June  2009,  you  should  opt-out  again  to  ensure  that  your  choice  is  saved  and 
recognized  by  our  ad  server."72  And  the  privacy  policy  of  [x+1]  states  "as  a  result,  if  you  opted 
out  of  targeted  advertising  by  [x+1]  prior  to  that  time  (either  through  [x  +  1]  or  through  our  opt 
out  listing  on  the  NAI  page),  your  choice  is  no  longer  effective."73  In  each  of  these  instances,  a 
user  who  had  opted  out  of  online  behavioral  advertising  from  one  of  these  companies  would 
have  that  opt-out  invalidated  even  before  the  opt-out  cookie  expired. 

Further,  while  NAI  members  are  not  required  to  provide  definitions  of  opting  out,  we 
found  some  instances  of  ambiguity  among  those  that  did.  The  privacy  policies  of  24/7  Real 
Media,  Glam  Media,  MAGNETIC,  and  Undertone  Networks  only  mention  opting  out  as 
pertaining  to  cookies;  we  assume  that  they  are  not  using  another  mechanism  for  tracking  users. 

We  observed  considerable  flux  and  instability  among  privacy  policies.  Perhaps  because 
of  the  August  2011  IAB  compliance  deadline,74  we  observed  22  NAI  members  changing  their 
privacy  policies  in  August  2011,  including  ten  that  changed  their  policies  in  the  last  week  before 
the  deadline.  At  least  28  NAI  members  self-reported  changing  their  privacy  policies  between 
January  1,  2011  and  July  31  2011;  nine  of  these  28  changed  again  in  August.  1-Behavior, 
InterCLICK,  Invite  Media,  Lotame,  and  Pulse360  explicitly  indicate  that  their  privacy  policies  may 
change,  and  ask  their  readers  to  return  for  updates. 


6  Discussion 

6.1  Limitations 

This  paper  checks  NAI  member  compliance  with  the  DAA  and  NAI  notice  and  choice  principles 
through  inspection  of  websites,  advertisements,  and  cookies.  However,  our  approach  has  some 
limitations. 

We  may  have  overlooked  some  notices  that  appear  outside  a  site's  privacy  policy. 
Neither  the  DAA  nor  the  NAI  explicitly  require  their  notices  to  be  placed  in  member  privacy 
policies.  However,  the  DAA  principles  indicate  that  notice  should  be  "clear,  meaningful,  and 
prominent."75  The  NAI  Principles  state  that  notice  is  to  be  given  "clearly  and  conspicuously."76 


http://www.akamai.eom/html/policies/privacy_statement.html#policy_opt_out 

71  http://www.dedicatednetworks.com/footer_privacy.asp 

72  http://www.undertone.com/privacy 

73  http://www.xplusone.com/privacy.php 

74  Supra  note  51 

75  Supra  note  10 

76  Supra  note  4 


Therefore,  when  we  are  unable  to  find  a  required  notice  on  a  member  privacy  policy  or  linked 
websites,  the  site  would  still  be  in  compliance  if  it  is  present  on  some  other  prominent  page  of 
the  website.  Nonetheless,  a  website  that  provides  a  notice  but  doesn't  link  to  it  from  its  privacy 
policy  is  arguably  not  communicating  clearly  and  conspicuously  with  its  users. 

We  were  unable  to  make  a  reliable  determination  about  which  observed 
advertisements  were  behavioral  and  which  third-party  cookies  were  associated  with  OBA.  We 
narrowed  the  scope  of  our  investigation  by  focussing  only  on  third-party  cookies  placed  by  NAI 
member  companies  and  by  eliminating  ads  that  we  judged  to  be  contextual.  However,  it  is 
likely  that  some  of  the  ads  and  cookies  we  eliminated  are  actually  subject  to  OBA  requirements. 
On  the  other  hand,  some  of  the  ads  and  cookies  we  included  may  not  actually  meet  the 
definition  of  OBA.  Nonetheless,  we  believe  our  dataset  provides  a  good  ballpark  estimate  of 
enhanced  notice  compliance  on  the  most  popular  websites,  and  we  provide  detailed 
information  about  our  methodology  and  findings  to  enable  readers  to  determine  the  basis  for 
our  compliance  estimates. 

6.2  Public  Policy  Implications 

The  results  of  our  study  raise  a  number  of  public  policy  concerns.  The  DAA  published  its 
principles  over  2  years  before  our  final  round  of  data  collection,  in  July  2009.  The  DAA  officially 
launched  its  self-regulatory  program  over  ten  months  ago  on  October  4,  2010. 77  Although  we 
have  observed  an  increasing  rate  of  compliance  in  the  weeks  leading  up  to  the  IAB  deadline, 
overall  compliance  has  been  slow.  We  observe  infrequent  compliance  with  the  "enhanced 
notice"  requirements,  and  only  18  of  the  74  NAI  members  indicate  DAA  membership  despite 
being  required  to  do  so. 

Beyond  shortcomings  in  notice  requirements,  the  DAA  and  NAI  opt-out  mechanisms 
contain  errors.  Opt-out  cookies  fail  to  be  set  for  some  members.  The  opt-out  cookies  for  others 
differ  between  the  two  mechanisms,  and  some  have  durations  shorter  than  the  required  five 
years. 

Even  if  the  opt-out  mechanisms  did  work  flawlessly,  they  do  not  adapt  to  changing 
membership.  NAI  membership  jumped  from  34  in  January  2010  to  66  in  February  2011,78  to  74 
in  August  2011.  A  user  who  has  opted  out  of  all  NAI  members  six  months  ago  would  not  be 
opted-out  of  a  dozen  members  today.  Further,  we  know  of  at  least  three  NAI  members  who 
were  acquired  and  ceased  to  operate  independently  during  the  duration  of  our  study:  aCerno, 
Dapper,  and  Tacoda.  This  raises  further  questions  about  whether  a  user  who  has  opted-out  of  a 
particular  company  needs  to  opt-out  again  when  such  an  acquisition  occurs. 

Given  the  focus  on  third-party  tracking,  users  are  unable  to  opt-out  of  tracking  by 
websites  they  are  currently  visiting  (e.g.,  companies  that  offer  both  first-party  content  and 
third-party  behavioral  advertising  services).  This  may  come  as  a  surprise  to  consumers  who 
think  they  have  opted  out  of  tracking  by  a  particular  company  but  may  not  realize  it  applies 
only  when  that  company  is  acting  as  a  third-party  behavioral  advertising  company.  The  DAA  and 
NAI  give  users  no  way  to  avoid  being  tracked  on  the  websites  of  NAI  members.  The  narrow 
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definition  of  OBA  proposed  by  the  FTC  and  adopted  by  the  DAA  and  NAI  may  be  insufficient  for 
addressing  consumer  privacy  concerns. 

We  also  observe  that  two  NAI  members  impose  limitations  and  demands  on  any  user 
who  visits  their  web  sites,  which  is  necessary  in  order  to  read  their  privacy  policies.  Undertone's 
privacy  policy  states  that  "by  using  the  Undertone  Site  Network,  this  website  or  sharing 
information  with  us,  you  give  your  agreement  to  this  Privacy  Policy."79  Undertone's  privacy 
policy  also  stipulates  limitations  of  liability.  Valueclick  Media's  privacy  policy  states,  "Please 
read  this  policy  carefully  since  by  visiting  this  website  ("Website")  and/or  sharing  information 
with  us,  you  agree  to  be  bound  by  this  Privacy  Policy."80  Valueclick  imposes  requirements  on  its 
users,  including  how  privacy  disputes  will  be  handled.  In  both  of  these  cases,  a  user  attempting 
to  learn  about  a  company's  behavioral  advertising  practices  and  read  the  notices  that  the  DAA 
and  NAI  require  will  be  struck  with  limitations  on  his  or  her  rights. 

It  is  worth  highlighting  the  flurry  of  compliance  improvements  we  observed  in  late 
August,  which  we  believe  are  in  response  to  the  lAB's  compliance  deadline.  The  IAB 
requirements,  found  in  the  IAB  Code  of  Conduct,  mirror  those  of  the  DAA,  with  an  added 
provision  for  enforcement.  An  IAB  member  found  not  to  be  in  comliance  with  the  Code  of 
Conduct  may  be  penalized,  having  its  IAB  membership  suspended.81  We  believe  that,  in 
addition  to  the  possible  threat  of  FTC  enforcement,  the  concrete  deadlines  and  enforcement 
proceedures  of  the  IAB  Code  of  Conduct  spurred  compliance. 

Finally,  we've  seen  that  a  number  of  NAI  members  provide  their  own  definitions  of 
opting  out,  going  beyond  the  minimum  bar  set  by  the  NAI  requirements.  This  is  positive  from  a 
privacy  perspective.  A  common  vocabulary  for  these  opt-out  variations  could  be  useful  for 
helping  consumers  understand  what  will  happen  when  they  opt-out. 
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